I dont know where the problem is. The crawler keeps hanging

Fix	Build a logger into all the places that are externally dependant. Never assume the external dependancies will behave themsleves.